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Executive Summary 



Over the past 20 years, the amount of information available to students 
via the Internet has increased dramatically. Access for students to 
technological resources used to locate information on the Internet has likewise 
increased. As a result, teachers are now being asked to teach students 
important 21 st Century Skills, including the ability to effectively evaluate 
website resources. 

During fall 2007, administrators and teachers from Sanford Junior High 
School collaborated on a research project with staff from the Center for 
Education Policy, Applied Research, and Evaluation at the University of 
Southern Maine aimed at enhancing students’ ability to effectively evaluate 
websites. Benchmarks for knowledge were created by project staff and were 
distributed to all science teachers. Those teachers then used the benchmarks 
to create their own content and methods for teaching the material. This project 
focused on instruction of students in 7 th and 8 th grade science classes because 
all of those students had access to their own laptop computer. Pre- and post- 
assessments were administered to all students participating in the project. 
Results revealed that students made improvements in their ability to evaluate 
website resources. 
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21 st Century Teaching and Learning: 

An Assessment of Student Website Evaluation Skills 
Introduction 

This report describes a collaborative research project undertaken by the 
Sanford (Maine) Junior High School science department and the Center for 
Education Policy, Applied Research, and Evaluation at the University of 
Southern Maine to help middle school students learn more effective website 
evaluation skills. Over the past 20 years, the availability of technological 
resources, especially via the World Wide Web, has increased in public schools 
across the country, encouraging teachers to continually adopt, adapt, and 
increase their use of those resources. As a result, teachers everywhere are 
being asked to teach students to critically evaluate websites using 2 1 st 
Century Skills, skills identified as those most important for success in the 
future. According to the Partnership for 21 st Century Skills, those skills 
include 1) knowledge in all content areas as well as in 21 st Century themes 
(global awareness; financial, economic, business, and entrepreneurial literacy; 
civic literacy; and health literacy), 2) learning and innovation skills, 3) 
information, media, and technology skills, and 4) life and career skills 
(Partnership for 21 st Century Learning, 2007). 

Making the need to teach children 21 st Century Skills yet more urgent is 
the number of classrooms in the United States that have computer access for 
some or all students. According to the U.S. Department of Education National 
Center for Education Statistics, in 2005, 94% of elementary schools had 
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Internet access with a ratio of four computers to every one student. Closer to 
home, the six-year-old Maine Learning Technology Initiative (MLTI) has 
afforded all public school students in 7 th and 8 th grades access to their own 
laptop computer, which in turn means all middle school-aged students now 
have access to a wide variety of website resources almost anytime. As a result, 
more attention is needed to help students to use these website resources 
wisely. 

As a result of the incredible amount of technology available in 
classrooms nationwide, there is an increased need for teachers and 
administrators to teach students how to accurately evaluate, comprehend, and 
judge the validity and reliability of resources located using the Internet. 
Technology is a strong catalyst for educational innovation and improvement; 
however, technology by itself does not act as the catalyst that drives learning. 
Today, information on the Internet, while readily available, is devoid of the 
evaluation process once provided by editors, publishers, and reviewers, and 
further by teachers and school librarians. Teachers, therefore, must not only 
learn and understand how new information is presented on the Internet, but 
they must also teach new concepts and approaches to help students 
comprehend and discriminate the content validity and reliability of information 
available there. This study was designed to determine the effectiveness of one 
approach used by Sanford Junior High School teachers to help their students 
acquire these Internet use skills. 

Much has been written about 21 st Century Skills in relation to students, 
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including how to teach those skills so students can accurately identify reliable 
and valid Internet information. However, limited information is available on 
how successful teachers are at adopting, adapting, implementing, and assessing 
those skills in their students within a ubiquitous environment. As a result of 
the lack of knowledge around teacher success, a pilot study at Skowhegan Area 
Middle School was created here in Maine in fall 2007 to try to begin to 
understand how students use information from the Internet when doing 
research, as well as to better understand how teachers go about teaching the 
skills necessary to locate information (Silvernail, et al., 2008) 

A sample of Skowhegan Area Middle School teachers along with several 
district technology integrationists, developed a curriculum strategy to help 
students learn how to locate and evaluate websites. The process was created 
for teachers instructing a sample of students in 6 th - 9 th grade. All teachers 
were given the same curriculum for the skills to teach around website 
evaluation, but they were asked to incorporate it into a topic area that matched 
what they were teaching in their own content curriculum. Though the period of 
time during which this study took place was short, the students did show signs 
of improvement in their ability to evaluate websites. 

Background 

During the summer and fall of 2007, Sanford Junior High School (SJHS) 
administrators and teachers began to take a closer look at how website 
evaluation was taught in their middle school. All agreed that website 
evaluation was an important skill and that because of its implications for 
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student learning, creating a process to standardize how all students in 7 th and 
8 th grades learn to critically analyze websites was an important school-wide 
goal to undertake. Because the school staff wanted to begin with a smaller 
group of teachers to ‘test’ the process and because all students are required to 
take a science class, the science department was identified as the group that 
would begin the initial work on this project. In addition, science teachers at 
SJHS expressed some problems they had experienced with students’ ability to 
evaluate websites, especially as a result of the significant amount of time 
students spend on their laptops finding current, scientific information. 

The goal for the SJHS science teachers was to implement website 
evaluation using technology in all science classes. Teacher characteristics 
were varied relative to teaching philosophy, amount of previous professional 
development, technology use, and materials and methods used to teach 
website evaluation skills to students. However, all the teachers felt relatively 
confident in their ability to use technology, valued the efficacy of technology, 
and were already using technology on different levels and at different rates 
within their classrooms. 

In the fall of 2007, Sanford Junior High School (SJHS) science teachers 
and administrators met with a research team from the Center for Education 
Policy, Applied Research, and Evaluation (CEPARE) at the University of 
Southern Maine (USM) to discuss the possibility of a collaborative research 
project to enhance students’ ability to evaluate websites. The interest from 
SJHS science teachers and administrators was not only on teaching students 
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how to evaluate websites appropriately, but to teach them a standardized set of 
principles to follow; the resulting effect being that students would be able to 
transfer the skills to other classrooms, especially once the skills were being 
taught school-wide. Due to a general consensus among SJHS staff to 
consistently teach 21 st Century Skills across science classes, the SJHS science 
teachers decided to collaborate on a project with CEPARE to extend the 
previous Skowhegan Middle School pilot study into a more extensive research 
project that would document their progress and the impact on student 
learning. Generally speaking, the goal of the project would be to integrate the 
21 st century skill of ‘evaluation’ into all science classes so students would be 
better positioned to comprehended Internet resources used for research. The 
teachers and administrators agreed that working with CEPARE on this project 
would give them the opportunity to create and test materials that could 
potentially be given to all teachers for use on all assigned research projects. 
This type of cross-curricular tool would allow students access to the same 
process in multiple content areas, increasing the likelihood that transference of 
skills would take place among students. 

Methodology 

Initial project planning meetings took place in October 2007. During 
those meetings, participating administrators, teachers, and CEPARE staff were 
brought together to discuss and plan the project. 

Goals of the Project 

The primary goal of this project was to help students learn how to 
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evaluate Internet resources in a systematic way, thus enhancing their ability to 
evaluate websites. In order to achieve this goal, a number of important actions 
were required by the project team. To start, teachers and researchers worked 
together to create benchmarks that would outline the concepts that 7 th and 8 th 
grade students at SJHS would need to learn in order to evaluate 
electronic/ digital resources within the context of authentic learning activities, 
specifically, science classrooms. In addition, project leaders and researchers 
worked together to help participating teachers effectively implement the 
benchmarks in their curriculum. Using the agreed-upon benchmarks, each 
teacher was asked to adapt or construct materials/concepts, determine 
frequency of use of those materials/concepts, and implement 
materials / concepts into their curriculums based on their own curricula 
agenda. 

Several other important steps were required in order to ensure not only 
that students acquired the appropriate skills, but also to make certain that the 
research project was carried out appropriately. A list of important activities 
follows: 

1) Benchmarks focused on website evaluation were developed for use 
by science teachers; website resources were provided to science 
teachers by project leaders but teachers were also encouraged to 
seek out their own (Appendix B) . 

2) Based on the benchmarks provided, teachers designed their own 
curricular materials using resources provided by project leaders or 
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on materials they located on their own. 

3) Sharing of information among science teachers occurred during 
weekly department meetings and via e-mail. 

4) Students were pre-tested before being exposed to the curriculum 
related to website evaluation and then post-tested afterward in 
order to determine the impacts of the curriculum intervention. 

5) Project leader Ms. Diana Allen conducted post-intervention 
interviews with all participating teachers to better understand the 
way the material was taught as well as how students reacted to the 
material. 

The team set a time frame to assess the students; December 2007 for the 
pre-assessment and June 2008 for the post-assessment. The period of time 
between October 2007 and December 2007 was used by individual teachers to 
develop independent project plans for the intervention. A post-intervention 
teacher interview was conducted by one of the project leaders. A more detailed 
project task list and timeline appears in Appendix A. 

Project Staff 

The SJHS science department consisted of a total seven science teachers 
in the pre-assessment group and six teachers in the post assessment group. In 
both pre- and post-assessment groups, the same three teachers taught 7 th 
grade and the same three teachers taught 8 th grade. One teacher taught both 
grades in the pre- and post-assessment group. Each class included varying 
student abilities and skill levels. Class size consisted of an average of twenty 
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students with a total of 25% of students overall identified as needing special 
education services. 

The project leaders at SJHS were Ms. Diana Allen, 7 th grade science 
teacher and Ms. Cindy Duggan, 7 th and 8 th grade science teacher and science 
department chair. Diana Allen coordinated meetings, communicated and 
interviewed science teachers, assisted CEPARE in assessment scoring, and 
served as the link between CEPARE and SJHS. Cindy Duggan provided 
assistance to Diana Allen as needed and assisted CEPARE in assessment 
scoring. 

Benchmarks 

As noted earlier, with the help of CEPARE, SJHS administrators and 
science teachers created a list of benchmarks for website evaluation. The 
benchmarks that were used by all teachers were as follows: 

• Students should be able to read a URL and gather certain information 
about the source: 

o Knowing the “value” of different domains, i.e. edu. (education site) 
or gov. (government site) 

o Are there personal names? Why is that good or bad? 

o Is the publisher one that is familiar and/or popular? Why is this 
relevant? 

• Students should be able to scan a page looking for certain “clues” to help 
them determine a page’s value: 

o Is the site current? Dated? 

o Who is the author of the page? Can they be contacted? 

o Are there links to additional sites, on the same topic? 

o Can I read and understand the information? Is it displayed in a 
way that is easy to use? 

o Does the information on the page apply to the research? 

o How many advertisements are on the page? 

o Is the information presented as facts or is someone trying to sway 
an opinion? 



9 




o Are there too many graphics and not enough information? Do the 
graphics apply to the topic? 

This tool was used by science teachers as a guide for what they would teach 
but not as a mandate for how they would teach it. In order to allow some 
amount of teacher autonomy, it was determined early on that they would 
determine how material would be taught. In addition to being useful to 
teachers, the benchmarks aided CEPARE in their creation of pre-and post- 
assessments for students. 

Assessments 

Both the pre- and post-assessments were constructed using a scenario- 
based format. Questions on the assessments revolved around accurately 
identifying and discriminating information presented on three websites. The 
research scenario asked students to plan a week’s worth of healthy menus by 
seeking out information online using three websites pre-determined by the 
research team. Students were directed to the three websites individually and 
were then asked to evaluate the usefulness, relevance, purpose, and reliability 
of each websites in relation to the task they had been given. The pre- and post- 
assessments were identical to ensure accurate before and after data. An 
explanation of the websites used for the pre- and post-assessment as well as a 
copy of the assessment instrument appear in Appendices C and D. 

The assessments were developed by CEPARE staff, and pre-tested for 
appropriateness and clarity in conjunction with the project conducted at 
Skowhegan Area Middle School. Several students from the pilot project school 
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were asked to take the assessment and were interviewed by the technology 
integrationist at that school to check for language difficulties and clarity of 
instructions. As a result of this student input, slight wording changes were 
made to the final version of the assessment and an alternate website #3 was 
selected to enhance differences between websites for student understanding 
and scoring purposes. A scoring rubric for the assessment was developed by 
CEPARE staff and the technology integrationist who helped create the 
assessment (Appendix E). 

Intervention 

As suggested, the curricular intervention materials were created 
primarily by individual science teachers respective to their grade level and 
content being taught at the time of intervention. The amount of time teachers 
spent providing the intervention to their students was determined by the 
teachers themselves and varied among teachers and grade levels. Except for 
the benchmarks, no specific guidelines were identified by the project team. 
Overall, teachers were encouraged to use individual resources or create 
materials in any topical area they deemed appropriate to their curriculum. 

The intervention was implemented by SJHS science teachers over 
approximately five months. Each science teacher started and ended the 
intervention at roughly the same time. The method of implementing the 
intervention generally followed one of two types of formats. The first format 
was in conjunction with an existing lesson. This involved all students looking 
at the same web page and discussing as a class the factors that contributed to 
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it being identified, according to the benchmarks, as a “good or bad” website. 
Instruction usually revolved around dissecting the site to reveal differences for 
research purposes. The second format was conducted in addition to an 
existing lesson. This consisted of the teacher assigning students a research 
project or topic and the students identifying and explaining the webpage layout 
in relation to the benchmarks. 

The pre- and post-assessments completed by SJHS students were scored 
by CEPARE project staff and two Sanford science teachers (project leaders). 
Student scores were based on values assigned using the rubric as a guide. At 
the start of each scoring session a sample of student tests were used to 
calibrate the ‘scorers’. To verify consistency in scoring, this process was 
repeated again after roughly half of the assessments had been scored. This 
process was conducted to obtain inter-rater agreement among scorers. For 
scoring of the pre-assessment, two CEPARE staff members and two SJHS 
science teachers scored each exam individually; student assessments were 
grouped randomly into sets of 30-45. For scoring of the post-assessment, one 
CEPARE project staff who scored the pre-assessment and the same two SJHS 
science teachers scored each exam individually; student assessments were 
grouped randomly into sets of 75-100. The results of the Sanford student test 
scores were normed and calibrated to the rubric. The evaluation methodology 
was shown to be effective in assessing design, content, and understanding by 
students. It should be noted that students who did not complete the survey 
were excluded from the analysis. 
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Results 



A summary of test results appears in Table 1. As shown in the table, 
results for SJHS revealed that the students performed well on the 

Table 1: Pre and Post 7 th & 8 th Grade SJHS Student Results 





Pre Assessment 


Post Assessment 


n 


mean 


std. dev. 


n 


mean 


std. dev. 


Students 


297 


15.01 


4.58 


347 


17.80 


5.59 



post-assessment in June 2008 when compared to the pre-assessment taken in 
December 2007. As may be seen in Table 1, SJHS students’ average scores on 
the post-assessment were above the pre assessment (17.8 vs. 15.0). In fact, 
statistical analysis of these results revealed there was a statistically significant 
improvement in student performance. Furthermore, analysis of the average 
scores, using Effect Size procedures, indicated students as a group improved 
their scores by 2/3 of a standard deviation. These Effect Size results suggest 
that the work SJHS science teachers did to prepare students for website 
evaluation as part of this project has substantially increased student skills in 
that area. Thus, the findings indicate the intervention was effective in 
improving students’ skills in evaluating web-based resources. Additional 
analyses of the data are available in Appendix F. 

Teacher observations 

Anecdotal observations from teachers regarding behavior and comments 
of students during pre- and post-assessments and during the intervention were 
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noted in a post-intervention interview. Valuable feedback was obtained 
regarding the intervention, assessments, and project impact on student 
learning. 

Two of the most interesting, and potentially useful pieces of feedback 
received from teachers were related to students’ understanding of the websites 
used for the pre- and post-assessments. Teacher observations during 
assessments noted that the students found the content of the websites to be 
useful and interesting; however, in some instances the questions were 
confusing for students, particularly those related to the third website. All the 
websites were found to be easy to navigate and understand by students. 
However, for both assessments, students expressed a desire to have websites 
reflect science content in relation to what they had studied. 

After completing the work with this project, SJHS science teachers 
expressed an interest in continuing website evaluation in their content area 
and on a school-wide level. The following suggestions were made by teachers: 

1 . Review grade level of materials . It is important that the content be grade 
and age appropriate. A review of materials may reveal needed 
modification to ensure that the assessment is more grade and age 
appropriate, as well as more content specific. 

2. Create a common vocabulary . Teachers felt that common vocabulary 
across all grades for the skills/ terms covered in the intervention was very 
helpful. 

3. Continuation of project . Teachers and students indicated that the 
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intervention was very useful to them in relation to their content area. 
Each teacher was encouraged to integrate website evaluation into their 
respective curriculum. 

4. Review the timing of the intervention . Introduce the skills early on in the 
school year so the skills are reinforced as the students engage in 
research activities for different content areas. 

Conclusions / Recommendations 

Conclusions 

The evidence gathered from this project suggests that on the whole, the 
project was successful. SJHS was able to demonstrate that by providing 
students with instructions for how to evaluate digital resources, students did 
improve their skills in evaluating online materials. Thus, it is concluded that 
the project was effective in demonstrating that the intervention could be 
effective in improving students’ 21 st Century Skills. 

All field-based research studies have limitations and this one is no 
exception. However, what may be considered limitations from attempting to 
implement a classic experimental research design in a school setting may 
indeed be considered strengths of this specific field-based research project. 
These include: 

- Teachers planning an intervention individually resulting in 
presentation of differing materials / intervention; 

- No professional development for teachers, or assessment of teacher 
skill levels allowing teachers to implement the intervention at their 
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level of understanding; 

- Teachers were allowed the freedom to determine frequency and alter 
intervention by adding or deleting resources, resulting in no 
standardized intervention; 

By allowing the freedom of development, process, and implementation of the 
project by teachers the impact on student learning was significant. 
Recommendations 

As a result of the research done as part of this project, CEPARE is 
prepared to make several recommendations for schools interested in using this 
model in the future. The recommendations are as follows: 

1) The model used for this project, whereby teachers were presented with 
benchmarks and charged with interpreting them and teaching them as 
part of existing curriculum was highly effective. Schools and school 
districts interested in enhancing students’ website evaluation skills 
should consider adopting the benchmarks such as those used here, but 
should ensure that those benchmarks are interpreted at either the school 
level or the teacher level in order that the learning be most meaningful to 
students. 

2) An integral part of the project conducted with SJHS was the leadership 
provided by administrators and project leaders from the school. Though 
much flexibility was allowed for teachers to use the benchmarks in their 
curriculum in a way that suited their students best, there was a project 
plan, a timeline, and a method for assessment (provided by CEPARE) 
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that was accounted for by the leadership. Schools considering adopting 
the benchmarks associated with this work should create a plan for 
implementation and assessment to ensure that teachers may determine 
clearly that students have achieved the desired learning outcomes. 

3) Teacher feedback regarding the flexibility they were allowed in teaching 
the material contained in the benchmarks was overwhelmingly positive. 
Because of the versatile nature of the benchmarks and teachers’ positive 
feedback, teachers and administrators providing instruction to various 
other ages and grade levels should consider adopting the benchmarks 
and tailoring them to the needs of their students. 

In summary, this pilot study has demonstrated the potential impact of 
interventions specifically designed to address 21 st Century Skills. 

Furthermore, the project has demonstrated the importance and feasibility of 
developing individual curriculum interventions tailored to specific content 
areas. Additional research is encouraged to replicate and possibly extend the 
findings from this pilot study. 
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Appendix A 



Project Timeline & Tasks: Sanford 



Task 


Key Participants 


Dates 


1 . Develop list of evaluation questions and 
objectives 


CEPARE 




2. Contact SJHS science teachers and 
administrator to participate in project 


CEPARE 




3. Create Assessment for pre- and post-test to 
measure the evaluation skills covered in the 
intervention (same assessment to be given 
for the pre and post test) 


CEPARE 




a. Select topic and web sites for 
assessment 


CEPARE 




4. Invite SJHS science teachers to group 

meeting to provide overview of project and to 
begin work. Items to review include 
timeframe, documentation, websites, and 
evaluation skills. 


CEPARE & SJHS 
science teachers 
and administrator 




5. Develop intervention. (SJHS science 
teachers). 


SJHS science 
teachers 




6. Administer assessment (pre) to 7 th & 8 th 
graders 


SJHS science 
teachers 


December 

2007 


7. Implement scoring rubric and score (pre) 
assessments. CEPARE and Project team to 
score Pre assessments 


CEPARE & Project 
Team 


Jan 2008 


8. Content Teachers /others at SJHS deliver 
intervention. (Content teachers to briefly 
document process for each class). 


SJHS science 
teachers 


February to 
June 


9. Re-administer assessment (post) to 7 th & 8 th 
graders 


SJHS science 
teachers 


June 


10. Conduct post-interview with teachers 

a. To gain an understanding of how 
their thought processes may have 
changed. 

b. Record teacher anecdotal 
observations of students during 
assessment (pre and post). Record 
teacher anecdotal observations of 
students during intervention 


Diana 


June 


1 1 . CEPARE and project team to score (post) 
assessments 


CEPARE & Project 
Team 


June 


12. Prepare final report 


CEPARE 


July/ Aug 
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Appendix B 



Supplemental Websites 

CEPARE website: 

http: / / www.msad54.org/weblinks / teacher / 2 1 stCentury.html 
SJHS supplemental websites: 

http: / / school.discoveryeducation.com/ schrockguide/eval.html 

http: / /www.oslis.org/ 

http: / / kathyschrock.net/abceval/ 

http:/ /www.multcolib.org/homework/webeval.html 

http: / /www. lib. berkeley.edu/TeachingLib/Guides/Internet/Evaluate.html 
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Appendix C 



Website Descriptions: pre/ post-assessments 

1. Website # 1 - The Egg Nutrition Center 
http:/ / www. enc-online.org/ 

Overview - This website was selected as a reliable and valid informational 
source. 

• URL - Identified as a .org 

• Information content / relevancy - Easy to read; clearly broken out; 
relevancy clear 

• Web Navigation - Clearly marked topic row; general introduction; new 
and interesting row of hyperlinks on the content (eggs) clearly marked on 
the page by topic 

• Dates - At the top of the page; clearly marked 

• Advertisements - None 

• Hyperlinks - Many; search button provided 

• Names of individuals/institutions - Clearly marked by research articles 

• Contact information - Clearly marked at bottom of page, with phone, fax, 
e-mail link 

• Bias - Stated as facts 

• Website goal - To inform and educate 

2 . Website #2 - Delightfulfood.com 
http:/ / www.delightfulfood.com/main.html 

Overview - A good website to entertain and provide information on the 
preferences of the individual but reliability to content is ambiguous, with no 
validity. 

• URL - Identified as a .com 

• Information content / relevancy - Websites/ hyperlinks clearly broken out 
by subject; overwhelming amount of hyperlinks provided; search button 
provided; relevancy ambiguous 

• Web Navigation 

o Clearly marked topic row, however many topics listed not dealing 
with food 

o general introduction wordy, with many subjects not relevant to the 
webpage 

• Dates - None 

• Advertisements - Several, broken out in a topic area 

• Names of individuals - Discussed as third person object 

• Contact information - Listed as “write to us” 

• Bias - Stated as opinions; not clearly presented 

• Website goal - Entertain; provide information 
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3. Website #3 - Nutrition for a Living Planet 
http:/ /www.diet-and-health.net/ 

Overview - Vague, only links provided. Website provides no reliable or valid 
information on home page. Website page provides links to valid and reliable 
sources of information by subject. Provides basic information by identification 
i.e. dictionary information but no valid sources cited in this section. 

• URL - Identified as a .net 

• Information content/ relevancy - Topic section listed; no search button 
provided; relevancy clearly linked to topic, however, subjects listed in 
section / topic area on home page few and ambivalent 

• Web Navigation - Clearly marked topic row, topics listed are not clearly 
identifiable; no general introduction 

• Dates - None 

• Advertisements - On all link pages; presented before information 

• Names of individuals/ institutions - Bibliography button provided; goes to 
cited research articles 

• Contact information - Listed as privacy policy/ contact us 

• Bias - None; stated information on health 

• Website goal - Provide information; all subjects listed revolve around 
health and diet 
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Appendix D 



Pre-and Post-Assessment Instrument 



Sanford ICT - Post-Assessment 

Introduction 



Dear Student: The sta^f at Sarford Junior H gh School is work ng to improve your 21st Certu-y Skills, specifical y 
tnose skills assoc ated with eva uating websites. In o-der to strengthen your Information Communication Technology 
(ICT) Literacy sk'lls, we have asked the Center fo- Education Po icy, Applied Researen, and Evaluation (CEPARE) at 
tne University of Southern Maine to help create a project that will offe' you the chance to improve your online 
researcn skills. 

Tne following assessment will nelp us to understand you r abiltes in tne area of eva uating the usefulness of online 
information. We will as< you to answer all the questions as tney re ate to eacn a~ three websites provided. None of 
tne information col ectec here will in any way afreet your grade or your relationship with tne Sanford Scnooi 
Department. You are not required to participate in th s study if you do not wish to. Your honest responses to the 
questions on th s assessment will be helpful and provide you the opportunity to contribute to tne current knowlecge 
about the process rer teaching eva uation skills using digital media and in so doing effect possible positive change to 
c assrooms throughout Sanford Jun o- High School. 

Tnere is no direct foreseeable risk for participating in this study. Your individual responses will be seen only by the 
evaluation team at CEPARE. Involvement in this study s competely voluntary. Reports will not include any 
information that will make it possible to dentify a participant. In any sort of quotation we will use, we will not 
include any nformation that will make it possible to identry the student being quoted. 

If you have any questions about this project you may reach the CEPARE evaluation team at (207)780-5044 or by 
ema I at cepare@usm.ma ne.edu. 

Tnank you for your willingness to participate in this important project. 

Sincere y. 

Ray Grogan, Assistant Prnc pal, Sanford Junior High School 
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Sanford ICT - Post-Assessment 

Untitled Page 



P ease enter you name, today's date, grade evel, and teacner's name in the spaces provided. 

First and Last Name 



Please enter today's date (for example, 03/02/2007) 

MM 00 YYYY 
Today's Date | / / 

What grade are you in? 



Teacher name 



Sanford ICT - Post Assessment 



Scenario: You have been assigned an Internet research project. The project is about nutrition, and your ass gnmert 
is to create a 7-day menu of nutritious, balanced meals. You will use information tnat you collect from the Internet 
to ceate your menu. 

Your task today: Look at each o* three websites (one at a time) and answer the survey questions about eacn site. 
You will be as<ed to evaluate eacn site and determine now useful t will be in cheating your 7-day menu. You may 
click through va^ous parts o c each website as you try to detemn ne i f it will be use*ul to you in bolding your 7-day 
menu. 

You will need to arswer the survey questions and look at the website at the same time. To do this, simply dick on a 
website to oper a new window. This will allow you to look at the survey questions n one window and the website 
you are evaluating in another window, 

You may need to resize and/or move botn windows in order to loo< at them at the same time. If you nave problems 
looking at the website and the survey at the same time, please ask you^ teacher for assistance. And remember. DO 
NOT CLOSE THE SURVEY WINDOW AT ANY POINT OR YOUR ANSWERS WILL BE LOST. You may, however, close the 
diPe^ent website windows. You must answer al of the quest ons on each page in orde- to move to the next page. 
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ICT Research Project 

Website #1 



Website #1 - The Eoo Nutrition Center 
click above link to access site 

Remember: your assignment is to determine if this website would be a good source of information for your 
project creating a 7-day menu of nutritious, balanced meals. 



1. How useful do you think this website will be for you in gathering information 
for your research paper? 

None or very little of It Is relevant (useful) to my topic 
Some of it is relevant (useful) to my topic 
All or almost all of it is relevant (useful) 



Who is the author and/or sponsor of this website? 



What is the MAIN purpose of the website? Are the authors trying to: 

Inform the reader 
Persuade the reader 
Entertain the reader 
Sell something to the reader 
Other (please specify) 



Why did you choose the answer above? 



£l 

Does the information in this website appear to be Opinion or Fact? 

All opinion 

Mostly opinion and some fact 
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ICT Research Project 



Mostly fact and some opinion 
Q All fact 

6. Why did you choose the answer above? 



7. How current (up-to-date) is the information contained on this website? 

Most/all of it is current 
Some of it is current 
Not current but still usable 
Not current and not usable 
Do not know 

8. Would the information on this website be considered primary source, 
secondary source or a combination of those? 

Primary source 
Secondary source 

Combination of primary and secondary sources 
Do not know 

9. List two sources used in this website (if you do not know what the sources 
are, please type "do not know" in the space provided). 

a) 
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Sanford I Cl - Post-Assessment 

Website #2 



Weosice »2 - DelichtPjlfood.com 

close website *1 ard clic< above link to access website *2 

Remember: yoir assignment s to determine if this website would be a good source o' information for your project 
creating a 7-day menu of nutritious, balanced meals. 

10. How useful do you think this website will be for you in gathering information for 
your research paper? 

None or very little of It Is relevant fuseful) to my toptc 
Some of It Is relevant (useful) to my topfc 
C) AN or almost all of It is relevant (useful) 

11. Who is the author and/or sponsor of this website? 

12. What is the MAIN purpose of the website? Are the authors trying to: 

Q Inform the reader 
(^) Persuade the reader 
Entertain the reader 
Sell somethin? to the reader 
(^) Other (please specify) 

I 

13. Why did you choose the answer above? 

a 

14. Does the information in this website appear to be Opinion or Fact? 

AM opinion 

Mostly opinion and some fact 
(~j Mostly fact and some opinion 
Q All fact 

15. Why did you choose the answer above? 

3 
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Sanford ICT - Post-Assessment 



16. How current (up-to-date) is the information contained on this website? 

£) Most/all of it Is current 
Some of It Is current 
Not current but still usable 
o Not current and not usable 
£) Do not know 

17. Would the information on this website be considered a primary source, a 
secondary source or a combination of those? 

o Primary source 
o Secondary source 

Combination of primary and secondary sources 
Do not know 

18. List two sources used in this website (if you do not know what the sources are, 
please type "do not know" in the space provided). 

«) 

b) 
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ICT Research Project 

Website #3 



Website #3 - Nutrition for a Living Planet 

close website #2 and click above link to access website #3 

Remember: your assignment is to determine if this website would be a good source of information for your 
project creating a 7-day menu of nutritious, balanced meals. 

19. How useful do you think this website will be for you in gathering information 
for your research paper? 

None or very little of It is relevant (useful) to my topic 
Some of it is relevant (useful) to my topic 
All or almost all of it is relevant (useful) 

20. Who is the author and/or sponsor of this website? 



21. What is the MAIN purpose of the website? Are the authors trying to: 

Q ^ Inform the reader 
Persuade the reader 
(^) Entertain the reader 

Sell something to the reader 
Other (please specify) 



22. Why did you choose the answer above? 

1 

£ 

23. Does the information in this website appear to be Opinion or Fact? 

All opinion 

Mostly opinion and some fact 
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ICT Research Project 



Mostly fact and some opinion 
Q All fact 

24. Why did you choose the answer above? 



25. How current (up-to-date) is the information contained on this website? 

Most/all of it is current 
Some of it is current 
Not current but still usable 
Not current and not usable 
Do not know 

26. Would the information on this website be considered primary source, 
secondary source or a combination of those? 

Primary source 
Secondary source 

Combination of primary and secondary sources 
Do not know 

27. List two sources used in this website (if you do not know what the sources 
are, please type "do not know" in the space provided). 

a) 
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Sanford ICT - Post- Assessment 

Comparison 



P ease cper eaci of :he websites you have looked at (linked below). The computer will allow you to have al of the 
websites open at the same time as wel as the survey. Look at each of the websites, either separately or al 
together, one more time and consider the foilowirg questions. 

Tie Ecq Nutrticn Carter 
Deliyhtfilfnr.ri.rn-n 
Nutrition *or a Livino Planet 

28. Which of the three sites you have reviewed would be most appropriate to use for 
your assignment? Why? 

3 

For the last 4 questions, you will not need to look at any specific websites. You should answer them as they relate to any type of 
Internet research. 

29. What is the best way to determine whether or not the information contained on 
a website is reliable (trustworthy)? 

3 

30. How can you determine whether or not a website is biased? 

31. What is the difference between a primary and a secondary source? 

a 

32. If you were assigned a research project and asked to find information on the 
Internet, please describe what a website would need in order to be valuable/useful 
in completing your assignment. 

a 
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Appendix G 



Assessment Scoring Rubric 
Website #1 

1. How useful do you think this website will be for you in gathering 
information for your research paper? 



1 point 


0 points 


• Some of it is relevant (useful) to 
my topic 


• None or very little of it is relevant 
(useful) to my topic 

• All or almost all of it is relevant 
(useful) 



2. Who is the author and/or sponsor of this website? 



1 point 


0 points 


• Egg Nutrition Center 


• Anything else 



3. What is the MAIN purpose of the website? Are the authors trying to: 



2 points 


1 point 


0 points 


• Persuade the 
reader 


• Inform the reader 


• Entertain the 
reader 

• Sell something to 
the reader 

• Other 



4. Why did you choose the answer above? (Examples of responses) 



2 points 


1 point 


0 points 


• Because the 

author attempts to 
persuade the 
reader into 
believing eggs are 
nutritious, 
delicious, & 
affordable 


• Website states that 
its target audience 
is egg lovers, egg 
producers / 
processors, and 
health care 
providers who 
want to learn more 
about how eggs 
contribute to a 
healthy diet 


• Because this 
website is for egg 
lovers and it is 
supposed to 
entertain them 



5. Does the information in this website appear to be Opinion or Fact? 
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2 points 


1 point 


0 points 


• Mostly fact and 
some opinion 


• Mostly opinion and 
some fact 


• All opinion 

• All fact 



6. Why did you choose the answer above? (Examples of responses) 



2 points 


1 point 


0 points 


• Because they are 
saying things that 
are true, but they 
also say what they 
think about the 
eggs 


• Because it mostly 
states what they 
think of eggs. The 
other part is fact 
because they’re 
trying to give you 
information on the 
subject so that 
you’ll get an 
interest and join 
their site 


• Because there 
aren’t any facts on 
this page 

• Because it has no 
opinions 



8. Would the information on this website be considered primary source, 
secondary source or a combination of those? 



1 point 


0 points 


• Combination of primary 
secondary sources 


• Primary source 

• Secondary source 

• Other/ do not know 



9. List two sources used in this website 

• 1 point for each listed credible source 

Website #2 

10. How useful do you think this website will be for you in gathering 
information for your research paper? 



1 point 


0 points 


• Some of it is relevant (useful) to 
my topic 


• None or very little of it is relevant 
(useful) to my topic 

• All or almost all of it is relevant 
(useful) 



11. Who is the author and/or sponsor of this website? 



1 point 



0 points 
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Janette Blackwell 



Anything else 



12. What is the MAIN purpose of the website? Are the authors trying to: 



2 points 


1 point 


0 points 


• Inform the reader 


• Sell something to 
the reader 


• Entertain the 
reader 

• Persuade the 
reader 

• Other 


13. Why did you choose the answer above? (Examples of responses) 


2 points 


1 point 


0 points 


• They are trying to 
inform the reader 
about resources 
for eating healthy 


• Trying to sell 
things like 
pictures to the 
reader 


• Because they are 
trying to get the 
reader to lose 
weight 


14. Does the information in this website appear to be Opinion or Fact? 


2 points 


1 point 


0 points 


• All opinion 


• Mostly opinion and 
some fact 


• All fact 

• Mostly fact and 
some opinion 



15. Why did you choose the answer above? (Examples of responses) 



2 points 


1 point 


0 points 


• There is a 

disclaimer on the 
side that states 
that it’s the 
opinion of the 
author 


• The author talks 
mostly about what 
she thinks but she 
also sites specific 
facts 


• Because it’s all fact 

• Because most of it 
is true 



18. List two sources used in this website 

• 1 point for each listed credible source 

Website #3 

20. Who is the author and/or sponsor of this website? 



2 points 



1 point 



0 points 
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• Author/ sponsor is 


• DietandHealth.Net 


• Other 


not listed 







21. What is the MAIN purpose of the website? Are the authors trying to: 



2 points 


1 point 


0 points 


• Inform the reader 


• Persuade the 
reader 


• Entertain the 
reader 

• Sell something to 
the reader 

• Other 


22. Why did you choose the answer above? (Examples of responses) 


2 points 


1 point 


0 points 


• Because they are 
informing you 
about what you 
can do to keep 
yourself healthy 


• To persuade the 
reader to make us 
eat better 


• They are trying to 
get you to think 
their product is 
good 


23. Does the information in this website appear to be Opinion or Fact? 


2 points 


1 point 


0 points 


• Mostly fact and 
some opinion 


• Mostly opinion and 
some fact 


• All opinion 

• All fact 


24. Why did you choose the answer above? 


2 points 


1 point 


0 points 


• There are facts 
about what you 
can do to stay 
healthy, and there 
is opinion about 
what foods and 
exercises are most 
effective 


• Because they have 
things that are 
suggested, which 
means that it isn’t 
complete fact, with 
mostly opinions 


• I choose that 
because it seems 
like all fact 

• I think it is all 
opinion because 
people were telling 
you things from 
their point of view 



26. Would the information on this website be considered primary source, 
secondary source or a combination of those? 



1 point 


0 points 


• Combination of primary 8& 


• Primary source 
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secondary sources 



• Secondary source 

• Other/ do not know 



27. List two sources used in this website 

• 1 point for each listed credible source 

Comparisons 

28. Which of these three sites you have reviewed would be most 
appropriate to use for your assignment? Why? 

• 1 point for listing a site 8& a credible reason for selecting that site 

29. What is the best way to determine whether or not the information 
contained on a website is reliable (trustworthy)? 

• 1 point for at least 1 credible method 

30. How can you determine whether or not a website is biased? 

• 1 point for at least 1 credible indicator of bias 
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Appendix F 



Additional Results Analysis 

Results from the pre-and post-assessments were analyzed using 
descriptive and inferential statistics. SPSS, a statistical program, and 
Microsoft Excel were used to obtain the data results. Analysis of the pre- and 
post-assessment scores indicated that the scores of students who received the 
intervention showed a small to medium increase in the Effect Size between the 
pre-assessment and the post-assessment for all 7 th and 8 th grade students. 
This information appears in Table 1 . 

Table 1: Pre and Post Student Assessment Results of Sanford Survey 





Pre Winter 
2007/2008 


Post 

Spring 

2008 


Total Number of Questions 


30 


30 


Total Number of Students 


297 


347 


Total points possible to earn by a 
student 


41 


41 


Total points possible to earn by all 
students (perfect score) 41xn 


12177 


14227 


Total points earned by all students 


4460 


6179 


% Students Correct 


0.37 


0.43 


Highest Student Score 


31 


32 


Mean Student Score 


15 (4460/297) 


17.8 

(6179/347) 


Mode Student Scores 


33 students 
obtained a 15 


26 

students 
scored a 15 


Medium Student Scores 


15 


18 


Standard Deviation 


4.58 


5.59 


Effect size 


0.61 


Range of test scores 


0 to 41 


0 to 41 
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Range of student scores 



4 to 31 



2 to 32 



Analysis between websites revealed differing student responses. On 
website #3, students demonstrated a small increase in four questions and a 
decrease across three questions. This may be due to the ambiguous content of 
website #3 which contained no salient markers in which to discriminate 
content, resulting in students’ inability to accurately evaluate content reliability 
or validity. Websites # 1 & #2 more clearly reflected content of the 
“Benchmarks for Website Evaluation,” making discrimination of valid and 
reliable information easier for students. In addition, the majority of student 
responses to questions about websites #1 & #2 showed that they could 
accurately identify how useful a website was for research, who the author is, 
and the main purpose of the website. 

Student results for website # 1 demonstrated an increase in their ability 
to identify how useful the information was, author/ sponsor, purpose, opinion 
or fact, and individual responses detailing information as to why they choose 
their responses. However, students demonstrated a decrease in accurately 
identifying primary, secondary, or both sources. Results suggest that students 
may need more instruction related to discriminating between a website that is 
used to inform and educate in relation to primary and secondary sources. 

For website #2, students demonstrated an increase in percentage from 
pre- to post-assessment results in the ability to accurately identify 
author/ sponsor, opinion or fact, primary/ secondary/ both sources, and more 
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detailed individual responses as to why they choose their answers. There was 
no increase in students’ ability to identify the purpose of the website and 
students demonstrated a decrease in their ability to discriminate the 
usefulness of website #2. It is interesting to note that this website is an 
entertainment website and more than 80% of students appropriately 
determined if it was a primary or secondary source but only 49% could identify 
if the website was useful for research purposes. This may suggest that 
students need more vigorous teaching in identifying and understanding data 
that is useful and relevant to research. 

Website #3 was the most ambivalent of the websites. This website was 
vague, provided only links, and had no reliable or valid information on the 
home page. Basic information was presented by links or by identification on 
other pages (e.g. dictionary information). Student results on the post- 
assessment for this website showed a small decrease in their ability to 
accurately name the author/ sponsor, identify the purpose of the website, and 
discriminate between fact and opinion. However, despite the ambiguousness of 
this website, students showed a small increase in their ability to accurately 
explain why they choose the site, identify primary and secondary sources, and 
list sources provided by the website. This may suggest that students may have 
difficulty discriminating information and need more instruction on website 
evaluation when no salient markers are present on a webpage. 

Overall, the scores do not reflect complete mastery of the skill - there is 
still a great deal of material that students do not fully grasp or transfer when 
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evaluating websites. When presented with websites that had information 
directly reflected in the benchmarks (e.g. dates, authors, domain), students 
could clearly evaluate and discern differences in and between websites and 
begin to determine the validity and reliability in relation to research. However, 
further analysis of the test results indicate that students at SJHS were not 
skilled at identifying and understanding ambiguous websites for research and 
could benefit from further instruction in this area. 
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